A JNI provider, so that JDK 17 can call the engine - #10

Panama is not on JDK 17 or 21, and those two are where most of the enterprise JVM ecosystem still is, so a client that only ships the Panama provider is a client half the users of Java cannot install. This adds a second provider over JNI, picked by the same service loader, so an application that adds both jars gets Panama on a recent JVM and this on an old one without choosing.

The shim

One C file, and it links against nothing. It opens libzu at run time and resolves what it calls, so building it needs a C compiler and the JDK headers and nothing else, and a shim built today opens a libzu built next year at the same ABI. Four decisions in it are worth knowing about.

It exports one symbol under the name JNI derives, Java_dev_zudb_jni_JniBinding_nRegister, and registers the other hundred and two from a static table. JNI_OnLoad cannot find a class that a module path or an application server's class loader holds, so the bootstrap has to start from a class rather than from the library.

Every string crosses as a byte[] rather than as a jstring. JNI's own conversions speak modified UTF-8 while the engine validates the real thing, so an emoji handed to NewStringUTF is a byte sequence the engine refuses. Java does the encoding and the decoding, and a string outside the basic multilingual plane is a test case.

A failure is built on the Java side from the status and the diagnostic fields, so the two providers cannot come to disagree about which exception a GQLSTATUS class names.

And the callbacks work, which is the part JNI makes hard. The engine calls progress on a thread the JVM has never seen; the shim attaches it as a daemon and detaches it on the way out, and the cookie that owns the global reference is freed when the connection closes or the watcher is replaced.

One suite, both providers

The cases that were in the Panama provider's test source set are now a module of their own, zudb-tck, and both providers run them. They are main sources rather than tests because a test source set is not a thing another module can depend on without a test jar and the trouble a test jar brings on the module path. One hundred and forty six cases run twice, so a difference between the two providers is a red build here rather than something a user finds. Which provider a run uses is named with -Dzu.provider, otherwise the two would race on priority and one of them would never be tested at all.

The Arrow reader runs its twelve twice as well. Handing a stream address to the engine is the one thing the two providers do differently enough that a reader working over one is no evidence about the other.

Five cases are JNI's alone, because they have nothing to say about Panama: a string outside the basic multilingual plane crossing intact, a column coming back as a window onto the engine's own memory rather than a copy, progress arriving from a thread the JVM has never seen, a frame refusing a heap buffer with the same wording the other provider uses, and a frame over direct memory being queried where it lies.

Building it

scripts/build-shim.sh builds the host platform and nothing else, because a C cross compiler for six other platforms is not something a laptop has. CI runs it on each runner of its matrix, and the release runs it on seven and collects the results into one jar. Maven never compiles C: a toolchain per platform is not something a Maven build can arrange and not something a user running mvn test should be made to have.

The shims profile is the check that a release is not a jar with one platform in it. It passes on the job that has downloaded all seven and nowhere else.

What CI does with it

A new jni job runs the suite on 17 and 21, on Linux and macOS, which is the whole point: a client that claims 17 and is only ever tested on 25 is a client that claims 17. It runs everything twice, the second time with -ea -esa, because a wrong length handed to NewDirectByteBuffer is the difference between a test that fails and a test that reads somebody else's memory.

Both native jobs now diff the vendored zu.h against the engine's before they build. The shim is written against that header and opens a libzu at run time, so a copy that has drifted compiles clean and then reads a struct that moved.

The release grows a shims matrix of seven, including two alpine containers for the musl pair and an -arch x86_64 cross build for intel macOS off the arm runner, and the release job downloads all seven before it packages.

Verifying the table by machine rather than by eye

A hundred and three native declarations against a hundred and three entries in a table is not something to check by reading. The entries were extracted from the C and the descriptors from the compiled class with javap -p -s, and the two lists were sorted and diffed to nothing.

Local: mvn install over the whole reactor is green. 41 in zudb, 147 in zudb-ffm, 151 in zudb-jni, 12 twice in zudb-arrow.